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Abstract 

Ontologies are knowledge representational structures which model domain knowledge in the form of concepts, 
roles, instances and their relationships. This knowledge can be exploited by an assessment system in the form of 
multiple choice questions (MCQs). The existing approaches which use ontologies expressed in the Web Ontology 
Language (OWL) for MCQ generation, are limited to questions of type "What is C?" or "Which of the following 
is an example of CT (where C is a concept symbol). Also, there are no systematic methods for generating incor- 
rect answers (distractors) from ontologies. Distractor generation process has to be given much importance, since the 
generated distractors determine the quality and hardness of an MCQ. In this paper, we propose two new MCQ gen- 
eration approaches, which generate MCQs that are very useful and realistic in conducting assessment tests, and the 
corresponding distractor generating techniques. Our distractor generation techniques, unlike other methods, consider 
the open-world assumption, so that the generated MCQs will always be valid (falsity of distractors is ensured). Fur- 
thermore, we present a measure to determine the difficulty level (a value between 0 and 1) of the generated MCQs. 
The proposed system is implemented, and experiments on specific ontologies have shown the effectiveness of the 
approaches. We also did an empirical study by generating question items from a real-world ontology and validated 
our arguments by domain experts. 
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1. Introduction because they require less administrative overhead and 

provide instant feedback to test takers. However, stud- 
Automated assessment systems serve as a method to ies by Barbara Qmss {m3) and sidick et al (1994) 

measure the mastery of learning as well as to provide show that developing effective objective test questions is 
a fast alternative for large scale assessment. Many tests time consuming and requires domain expertise to gen- 
like TOEFL, IELTS, ORE, GMAT are dependent on on- erate good quality MCQs So> there ifj a need for an 
line assessment systems to make the assessment task 



imc a^c^mcm ^Lcm, L u ma^c uic ^cmc.u automated method for MCQ generation from a given 

easier. Such systems mainly use multiple choice ques- knowledge source 
tions rather than subjective questions for conducting the 

j es j Recently, a handful of studies ( Alsubait et al. (2012) 

tt • iv/r ^ t ■ i r^u ■ /-> ..• /\*r^r\ \e Cubric and Tosic (2010); Papasalouros et al. (2008) 
Using Multiple Choice Questions (MCQs) for assess- v " F v ' 

t , , J - t , , ■* \r. f , M.Tosic and M.Cubric (2009); Al-Yahya (2014) 

ments has both merits and demerits. They are preferred v „ 7 ' : v ' 

for assessing broad range of knowledge. This is mainly Zoumpatianos et al. (201 1); Aitko et al. (2009)) ex- 

plored the use of structured domain knowledge in the 

form of description logic based ontologies to automati- 

-Con-esponding author cally generate MCQs. This would enable online assess- 
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ontologies. Some of the challenges that the existing ap- 
proaches try to address are: (i) How to frame interesting 
and good quality questions from ontologies? (ii) How 
to generate proper incorrect answers (distractors) for a 
generated question? (iii) How to control the difficulty 
level of the generated questions? Although the previous 
efforts were not in vain, there are substantial shortcom- 
ings in fully exploiting the formalized knowledge in an 
ontology for MCQ generation. In this paper, we show 
that a better understanding of the semantics of a given 
ontology (expressed in Web Ontology Language), the 
three challenges can be addressed more elegantly. 

Challenge 1. Framing interesting and good quality 
questions. In the literature, the approaches which use 
ontologies generate MCQs of the form: "What is C?" 
or "Which of the following is an example of concept 
C?" ( where C is a concept symbol ). These questions 
are of very basic type (Bloom and Krathwohl (1956)) 
and are not much useful for asking questions with do- 
main related specifics. The approaches which generate 
such questions, do not appropriately make use of the ax- 
iomatized knowledge in the ontology. Furthermore, re- 
strictions (existential, universal and cardinality) on con- 
cepts and roles in ontologies are not utilized properly for 
question generation in any of the current approaches. 

Consider a movie ontology with statements, 
Movie (braveHeart) 

MovieDA = Movie n BisDirectedBy . Acto]^^ 
MovieDA (braveHeart ) 

With respect to this, we can frame a question about the 
instance braveHeart: "Choose a movie directed by an 
actor?" Our approach in this paper is an effort in this 
direction. 

Challenge 2. Proper distractor generation. Under the 
closed-world assumption (CWA), we can choose any in- 
stance which is different from the instance braveHeart 
as a distracting answer. But, the semantics of Web On- 
tology Language (OWL) adheres to the open-world as- 
sumption (OWA): statements which are not logical con- 
sequences of a given knowledge base are not necessar- 
ily considered false. Therefore, not all distractors which 
are generated under CWA can be guaranteed as true dis- 
tractors. 

We observed that most of the existing MCQ genera- 
tion techniques (Al-Yahya (2014)) randomly select in- 
stances which do not belong to the class of the correct 
answer as distractors. The incorrectness of the distrac- 
tors cannot be ensured by this random selection method, 
which in turn made it necessary to manually check the 
correctness of the question items before making use of 



them. We address this issue by proposing a systematic 
method to generate only those question items which are 
valid under OWA. 

Challenge 3. Control the difficulty level of the gener- 
ated MCQ. MCQs of varying difficulty level are nec- 
essary to assess the depth of knowledge of a learner 
(student). We introduce a difficulty measure to find 
out the hardness of the generated MCQs based on the 
similarity-based theory suggested by Alsubait et al. 
(2013). 

In this paper, we propose two approaches (i) node- 
label-set based approach (ii) edge-label-set based ap- 
proach to generate (two) interesting types of MCQs. We 
adopt description logic specifications of the ontology 
to generate the so called label-sets (node-label-sets and 
edge-label-sets). A measure to estimate the difficulty 
level of generated MCQs is also proposed by means 
of these label-sets. We study the feasibility of our ap- 
proaches by implementing them and generating MCQs 
from some sample ontologies. To validate our new ap- 
proaches and difficulty measure, we generated question 
items from a real-world ontology and got them evalu- 
ated by domain experts. Statistics of our empirical eval- 
uation validate our arguments and are detailed in Sec- 
^ tion 6. The new notations and abbreviations that we 
introduced in this paper are listed in Appendix B along 
with their meaning. 

2. Preliminaries 

In this section we describe MCQ, the Description 
Logic (DL) SHIQ. based ontologies (SfilQ ontolo- 
gies) and an example ontology (Harry-Potter-Book on- 
tology). 

2.1. Multiple Choice Questions 

We adopted a simple and general definition of 
MCQs (Alsubait et al. (2012)) to explain our approaches 
and observations. 

Definition 1. A multiple choice question MCQ is a tool 
that can be used to evaluate whether (or not) a student 
has attained a certain learning objective. It consists of 
the following parts: 

Stem (S). The statement that introduces a problem to 
the student. 



Choices. A set of options A = { A U A 2 , A„ 
It can be further divided into two sets: 



,m > 2. 
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Name 



Table 1 : The syntax and semantics of S'HIQ concept types 
Syntax Semantics 



Table 2: The syntax and semantics of S'HIQ ontology axioms 



Name 



Syntax Semantics 



atomic concept A 

top concept T 

bottom concept _L 

negation ->C 

conjunction C n D 

disjunction C U D 

existential restriction 3R.C 

universal restriction VR.C 

min cardinality > nS.C 

max cardinality <mS.C 



A 1 (given) 

A 1 

<P 

A J \C J 

C J nL» J 

C J uD J 

{xeA 1 \3y.(x,y)eR T AyeC 1 } 
{ x e A 1 | Vy.(x,y) e R 1 => y e C 1 ) 
{x e A 1 | #{y | {x,y) eR T }>n} 
[xeA 1 \ #{y\{x,y)eR I }<m} 



role hierarchy R E S R 1 C S 1 

role transitivity Tran(fl) R 1 o R 1 c fl J 

TBox concept inclusion CEO C 1 Q D 1 

concept equality C = D C 1 = D 1 

concept assertion C(a) a 1 e C 1 

ABox role assertion R(a,b) (a I ,b I )eR I 

inequality assertion a i> b a 1 + b 1 



a. 



b. 



Key. A set of correct 
{ A\, Ai, ...,Ai }, 1 <i<m. 



options K = 



Dis tractors. 

[A i+U ...,A„ }. 



A set of incorrect options D = 



Note : In this paper we assume K as a singleton set. 
We fix the value of m, the number of options, in our 
experiments as 4, as it is the standard practice in MCQ 
tests. 



2.2. S'HIQ DL and S'HIQ Ontologies 



The Description logic S'HIQ is based on an ex 
tension of the well-known logic J[£C (Schmidt- 
SchaubBand Smolka (1991)), with added support for 
role hierarchies, inverse roles, transitive roles, and qual- 
ifying number restrictions (Horrocks et al. (2000)). 

We assume jV c and N R as countably infinite sets of 
atomic concepts and atomic roles. A S'HIQ role is ei- 
ther r e Nr or an inverse role j*t with r e Nr. When 
R = r, we denote the inverse of the role R by R — r ~. 

The set of concepts in S'HIQ is recursively defined 
using the constructors in Table- 1, where A e Nc,C,D 
are concepts, R, S are roles, and n,m are positive in- 
tegers. A S'HIQ ontology is a set (denoted as O) of 
axioms of the type specified in Table-2. A role R in O is 
transitive if Tran(/?) e O or Trw{R ) e O. Given an O, 
let R\ E R2 be the smallest transitive reflexive relation 
between roles such that R\ E R2 e O implies R\ Eo R2 
and R~ E 0 R^- For a S'HIQ ontology O, the role S 
in every concept of the form > nS.C and < mS.C in O, 
should be simple, that is, R Eo S holds for no transitive 
role R (Baader et al. (2003)). 

The semantics of S'HIQ is defined using interpreta- 
tions. An interpretation is a pair J = (A J , /) where A J 
is a non-empty set called the domain of the interpreta- 
tion and / is the interpretation function. The function / 



assigns a set A 1 c A J to every A e N c , and assigns a re- 
lation r 1 c A J x A J to every r e Nr. The interpretation 
of the inverse role r~ is r~ := \{x,y) \ (y,x) e r 1 ^}. The 
interpretation is extended to concepts and axioms ac- 
cording to the rightmost column of Table- 1 and Table-2 
respectively, where #V denotes the cardinality of a set 
V. 

We write I |= a, if the interpretation I satisfies the 
axiom a (or a is true in the interpretation). J is a model 
of an ontology O (written I \= O) if J satisfies every 
axiom in (9. If we say a is entailed by O, or a is a 
logical consequence of O (written O \= a), then every 
model of O satisfies a. A concept C is subsumed by Z) 
w.r.t. O if (9 |= C E D, and C is unsatisfiable w.r.t. (9 
if (9 |= C E -L. Classification is the task of computing 
all subsumptions A E B between atomic concepts such 
that A,B e A^ c and O |= A E B; similarly, property 
classification of (9 is the computation of all subsump- 
tions between properties R E S such that S e Nr 
and (9 |= R E 5 . By inferred ontology (written O \= 
O'), we mean the ontology after class classification and 
property classification, with all the newly entailed role 
and concept assertions. 

2.3. Harry-Potter-Book Ontology 

We use a synthetic ontology called Harry-Potter- 
Book ontology (O) as a running example in this paper. 
This ontology, which models characters in Harry Potter 
story 1 , was developed by knowledge engineers from our 
research team. Table-3 and Table-4 show the required 
portions of the TBox and ABox of the ontology respec- 
tively. 

Some of the explicit knowledge formalized in 
the ontology is explained below. Harry Potter, 
Hermione Granger, RonWeasley, Draco Malfoy and 
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Table 3: The TBox axioms of Harry-Potter-Book ontology 



(1) HogStudent = Student n VhasPet . Creature n BhasPet.Pet n< lhasPet . Creature 

(2) Pet = Creature n VisPetOf .HogStudent n BisPetOf .HogStudent (3) HogStudent E Student 



(4) Student E Human (5) Owl E Pet 

(7) Toad E Pet (8) Pet E Creature 

(11) HogStudent E Gryffindor U Slytherin 

(13) Muggle n Wizard E ± (14) Pet n Student E ± 

(16) Half blood n Muggle E _L (17) Pureblood n Muggle E ± 

(19) Gryffindor n Slytherin E ± (20) Owl n Toad E -L 

(22) Owl n Cat E ± (23) Toad n Rat E ± 

(25) Rat n Cat E ± (26) hasFriend E knows 



(6) Rat E Pet 

(9) Muggle E Human (10) CatE Pet 

(12) DrumstrangStud n HogStudent E ± 

(15) Pet n Human E -L 

(18) Pureblood n Halfblood E -L 

(21) Owl n Rat E ± 

(24) Toad n Cat E ± 

(27) hasHelped E know^^v* 



shanks, a cat. 



2.4. Label-sets 



Neville Longbottom are students of Hogwarts School. 
All Hogwarts students should have exactly one Crea- 
ture as Pet (axiom (1) of Table-3). Owl, Toad, Cat and 
Rat are the common pets (they are disjoint concepts) 
(axioms (20-25)). 

We have concepts like Muggles, Human, Wizard, 
Pureblood, Halfblood etc in our ontology. Muggles are 
not Wizards (axiom (13)). All Muggles are Humans 
(axiom (9)). Pureblood, Halfblood and Muggle are dis- 
joint concepts (axioms (16-18)). Hogwarts students be- 
long to either Gryffindor house or Slytherin house (ax- 
iom (11)). Harry Potter owns an owl named Hedwig as^ 
his pet. RonWeasley's pet Scrabbers is a rat. A toad 
named Trevor is Neville's pet. Hermione's pet is Crook- 

To generate question items corresponding to each in- 
stance and each pair of instances of an ontology, we 
associate a set, containing the constraints they satisfy, 
along with each of them. We call these sets as label- 
sets in general. Label-set of individual instance is called 
node-label-set and label-set of a pair of instances is 
called edge-label-serr 

Node-label-set. The label-set of an instance is 
the set which contains the class expressions and 
(existential, universal and cardinality) restrictions 
satisfied by that instance. It is represented as 
£o(x), where x is an instance in ontology O. 
For example, the node-label-set of the instance 
harryPotter from O = Harry-Potter-Book ontology, 
is given as: .^(harryPotter) = { HogStudent , 
Student, Human, Wizard, Halfblood, 
Gryffindor, 3hasPet.Pet, BhasPet . Owl , 
VhasPet . Creature , <lhasPet . Creature }. 



Edge-label-set. The label-set of a pair of instances 
(x,y) is the set that contains all the property re- 
lationships (role names) from the first instance 
to the second instance. It is represented by 
£o(x,y). From our example ontology O, edge-label- 
set of the pair (harryPotter , hermioneGranger) is 
given by: .^(harryPotter , hermioneGranger) = { 
hasFriend, hasHelped, knows }. 

2.5. Label-set Generation techniques 

Node-label-set Generation. We generate the label-set 
of an instance x from an ontology O by first creating the 
corresponding inferred ontology O' (using a reasoner). 
From O' we find all the atomic concepts {named con- 
cepts) and (existential, universal and cardinality) restric- 
tions satisfied by the instance as follows: 

Step 1: All the named concepts which are satisfied by 
x are obtained by the following SPARQL query: 

©prefix ex: <http://example.org> 
select distinct ?concept 
where { ex:x a ?concept } 

For example, { HogStudent , Student , Human , 
Wizard, Halfblood, Gryffindor) is the set of 
named concepts which corresponds to the instance 
harryPotter. 

Step 2: In order to get the restrictions satisfied by 
x, we access the class definitions of the concepts which 
are obtained in the first step and then consider the exis- 
tential, universal and cardinality restrictions on the right 
hand side of the class definitions to form the label-set. 
We also include the stand alone restrictions of the in- 
stance, which are not a part of any class definitions, in 
the label-set. 

Continuing with our example of the instance 
harryPotter, enrichment of the label-set is done by 
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Table 4: The ABox axioms of Harry-Potter-Book ontology 

DrumstrangStud(viktorKrum) 
HogStudent (harryPotter) 
HogStudent (hermioneGranger ) 
HogStudent (nevilleLbottom) 
HogStudent (dracoMalf oy) 
HogStudent (ronWeasley) 

Gryf f indor (ronWeasley) 

Gryf f indor (harryPotter) 

Gryf f indor (hermioneGranger) 

Gryf f indor (nevilleLbottom) 
Muggle (hermioneGranger) 

Pureblood (dracoMalf oy) 

Pureblood (ronWeasley) 
Half blood (harryPotter) 
Weasley (ronWeasley) 

BhasPet . Cat (hermioneGranger) 

BhasPet .Rat (ronWeasley) 

BhasPet . Owl (harryPotter) 

BhasPet .Toad (nevilleLbottom) 

BhasPet .Toad (dracoMalf oy) 
Toad(trevor) 

Owl (hedwig) 

Owl (errol) 
Cat ( cr ookshanks ) 
Rat (scrabbers) 

Slytherin (dracoMalf oy) 

Slytherin(tomRiddle) 
hasPet (harryPotter , hedwig) 
hasPet (nevilleLbottom, trevor) 
hasPet (ronWeasley , scrabbers) 
hasPet (hermioneGranger , crookshanks ) 

Wizard (viktorKrum) 

Wizard (harryPotter) 

Wizard (dracoMalf oy) 

Wizard (tomRiddle) 

Wizard (nevilleLbottom) 
knows (nevilleLbottom, harryPotter) 
knows (harryPotter , tomRiddle) 
knows (dracoMalf oy, harryPotter) 

hasFriend (harryPotter, hermioneGranger) 

hasFr iend (harryPotter , ronWeasley) 
hasHelped (harryPotter .hermioneGranger ) 
errol * hedwig 



VhasPet . Creature , <lhasPet . Creature }. The 
node-label-sets of some of the other instances in the 
Harry-Potter-Book ontology are given in Table-5. 

We maintain the disjoint concepts of each of the 
named concepts in the label-set JLo{x) as a set called 
Disjoint (£o(x)). From our example ontology, Dis- 
joint (^(harryPotter)) is { Pet, Owl, Rat, 
Toad, Cat, Muggle, Pureblood, Slytherin }. 
We later use Disjoint sets of label-sets in Section 3.1.3 
for generating distractors. 

Definition 2. Disjoint (£o(x)) = {z)|<9|=(DnC = 

_L) and C e X.o(x) and D is an atomic concept in O } 

Edge-label-set Generation. The edge-label-set of a pair 
of instances (x, y) can be easily generated from O' using 
the SPARQL query: <>y 

©prefix ex: <http://example.org> 
select distinct ?role 
where { ex:x ?role ex:y } 

We getgS^(harryPotter , hermioneGranger) = { 
hasFriend, hasHelped, knows } from the above 
query by substituting x = harryPotter and y = 
hermioneGranger. Other necessary edge-label-sets 
^ Tised in this paper are given in Table-8. 



tial, un 



obtaining (i) existential, universal and cardinality re- 
strictions on each of the atomic concepts in the label-set 
(i.e., the set { BhasPet . Pet , VhasPet . Creature , 
<lhasPet . Creature } ) and (ii) other restrictions 
on the instance (i.e., { BhasPet . Owl } ). We can 
use the Jena Ontology API 2 or the OWL API 3 for 
extracting such restrictions from ontologies expressed 
using the Web Ontology Language (OWL). Fi- 
nally, we get .^(harryPotter) as { HogStudent, 
Student, Human, Wizard, Halfblood, 
Gryf f indor, BhasPet . Pet , BhasPet. Owl, 



2 http://jena.apache.org (last accessed 11th Dec 2014) 
3 http://owlapi.sourceforge.net (last accessed 1 1th Dec 2014) 



3. Proposed MCQ Generation Approaches 

Once we get the label-set of all instances (node-label- 
set) and label-set of all the pairs of instances (edge- 
label-set) in the ontology, we can generate the MCQs 
based on the following two approaches. 

3. 1. Node-label-set Based Approach 

In this approach, we frame a question based on the 
label-set of an instance (with that instance as the key). 
As a first step, the DL related terminologies in a label- 
set need to be processed for making it suitable to frame 
a human understandable stem. We achieve this by the 
following Label-set-Reduction process. We also discuss 
our distractor generating method, and a measure to com- 
pute the difficulty of the generated MCQ in detail below. 

3.1.1. Label-set-Reduction 

Removing named super concepts. Consider the node- 
label-sets in Table-5, we can see that they have all the 
named concepts which contain the corresponding in- 
stances. Some of these concepts might be related hi- 
erarchically in the ontology, resulting in redundant in- 
formation in the label-set. For example, consider the 
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Table 5: Node-Label-set of instances in Harry-Potter-Book ontology 



Xo(harryPotter) 

Xo(ronWeasley) 

Xo(hermioneGranger) 

X(dracoMalf oy) 

X(hedwig) 
X(errol) 
X(scrabbers) 
X(trevor) 
X(crookshanks) 
X(nevilleLbottom) 

X(viktorKrum) 
X(tomRiddle) 



= { HogStudent , Student, Human, Wizard, Half Blood, Gryffindor, 

BhasPet . Pet , BhasPet . Owl , VhasPet . Creature , <lhasPet . Creature } 

= { HogStudent , Student, Human, Weasley, Gryffindor, Pureblood, 
BhasPet. Pet, BhasPet. Rat, VhasPet . Creature , <lhasPet . Creature } 

= { HogStudent , Student, Human, Muggle, Gryffindor, BhasPet. Pet, 
BhasPet . Cat , VhasPet . Creature , <lhasPet . Creature } 

= { HogStudent , Student, Human, Slytherin, Pureblood, BhasPet. Pet, 
BhasPet. Toad, VhasPet . Creature , <lhasPet . Creature } 

={0wl, Pet, Creature, VisPetOf . HogStudent , BisPetOf . HogStudent } 

= { Owl , Pet , Creature } 

={Rat, Pet, Creature, VisPetOf . HogStudent , BisPetOf . HogStudent } 

= {Toad, Pet, Creature, VisPetOf . HogStudent , BisPetOf . HogStudent } 

= {Cat, Pet, Creature, VisPetOf . HogStudent , BisPetOf . HogStudent } 

= { HogStudent , Student, Wizard, Gryffindor, BhasPet.Pet, 
BhasPet.Toad, VhasPet . Creature , <lhasPet . Creature } 

= { Wizard, DrumstrangStud } 

= ' Wizard, Slytherin } 



label-set i^(harryPotter), it contains the concepts set of an instance x in ontology O, "Rnls is represented 
HogStudent and Student. Since it can be inferred as "R(£o(x)). 
from the concept HogStudent that harryPotter|jSi> 



dant information in the label-set. We remove such re- 



also a Student, we can say that Student is a redun- 

luch i 



dundant information by finding out the relationship be- 
tween concepts in the label-set and excluding all the 
named concepts which subsume at least one named con- 
cept in the label-set. Therefore, from the label-set of 
harryPotter, the concepts Human and Student can 
be removed because of the TBox axioms (4) and (3) (in 
Table-3) respectively. 

Definition 3. Let S c £o(x) be a maximum subset of 
named concepts in Hoi x ) sucn that V u e S, 3 v e £o(x) 
and v\Zu. Then, label-set after reduction is £o(x)\S. 

One reason for this reduction is that we are doing rea- 
soning on the ontology prior to generating the label-sets. 
Also, while authoring an ontology, it is common to have 
redundant instantiation of concepts which are hierarchi- 
cally related. 

The next step in the reduction process of the label-set 
is the reduction of terms with quantified and cardinality 
restrictions. We propose a set of reduction rules (in the 
next section) to achieve this task. We call the new node- 
label-set after the two reduction steps as the Reduced- 
node-label-set (Rnls, for short). In terms of node-label- 



Reduction rules. To extend the Label-set-Reduction to 
incorporate quantified and qualified cardinality expres- 
sions, we have formulated a set of reduction rules (Ta- 
ble 6). These rules are applied by considering pairs of 
restrictions at a time from the label-set. These rules are 
meant to do reasoning at instances level such that the DL 
based constraints in the label-set can be converted into 
a more human understandable representation. Pairs that 
are not shown in Table-6 cannot be reduced and hence 
they are used as they are in the 'Rnls. 

We introduce two notations 3! and 3 in our reduc- 
tion rules in Table-6. 3! in 3\R.C is defined to be 
3R.C n <IR.C. For example, 3!hasPet.0wl de- 
notes that there exists exactly one hasPet relation to 
the concept Owl. 3 in 3R.C is used to represent the 
pair (Bfl.C, Vfl.C). i.e, 3R.C = 3R.C n VR.C (i.e., 
BisPetOf .HogStudent and VisPetOf . HogStudent 
is represented as 3isPet0f . HogStudent). If the re- 
strictions like 3R.C and W?.C appear together in a label- 
set, the former guarantees the presence of an edge R and 
makes it necessary that the latter should be satisfied by 
the condition other than vacuously true case. This ob- 
servation is useful when we frame the question state- 
ment (stem) of an MCQ. 
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Table 6: Reduction rules 



Phase No. Rule No. Restriction- 1 Restriction-2 Condition 



Result 



1 


1 


3R U .C U 


3R V .C V 


C u E C v A 


^« E -Ry 






2 




VR V .C V 


C u E C v A 


E /?« 


VR U -C U 


2 


3 


VRu-Cu 


VR V .C V 


C v E C u A 


= Ru 


VRy.Cy 




4 


VR U -C U 


VR V .C V 


C v E C M A 


R u c /?v 


VR U .Cy,VRy.Cy 




5 


3R U .C U 


VRy.Cy 


R u = R v a 


r = r 


$Ru-C u 




6 


VRu-C„ 


3R V .C V 


Cy C C„ A 


R« E /? v 


3R U .C U , 3Ry.Cy 




7 


VRyCu 


3R V .C V 


C„ E C v A 


E R u 


13R U .C U , V*R V .C U 


3 


8 


VR U C U 


3R V .C V 


C 1( E C v A 


R U \Z Ry 


VR tt .C U , 3Ry.Cy 




9 


3R U .C U 


> nR v .C v 


C v E C 1( A 


RyCR u A n > 1 ^Jifiv.C,, 
R„ E R v A n u > n v > n u R u .C u 




10 


> n u R u .C u 


> n v R v .C v 


C u E C v A 




11 


(> n)3R u .C u 


< nR v .C v 


C u E C v A 


R u E R v A n = 1 


3\R U .C U 


4 


12 


(3)3fl„.C„ 


3\R V .C V 


C M E C v A 


fl„ E fl v _^ 

fl„ E fly 


3\R U .C U 




13 


(3)3/?„.C„ 


3\R V .C V 


C v E C u A 


3\R U .C V 


5 


14 




3\R V .C V 


C M E C v A 


E /?y 


3\R U .C U , 3\R V .C U 




15 


3\R U .C U 


3\R V .C V 


Cy E C„ A R u E /?„ 


3\R U .Cy, 3\Ry.Cy 



The procedure Generate-Reduced-Node-Label-Set si 
illustrates the reduction process. Line 2 corresponds to 5 /V: 
the first part of the reduction process, where the proce^ 
dure calls Named-Super-Concepts to find all the named 
concepts which subsume at least one named concept in 
the label-set. Lines 3-10 illustrate how the reduction 
rules are applied on a label-set. These reduction rules 



are applied in five phases. Each phase (denoted as P) 
handles the reduction of carefully chosen restriction pat- 
terns whose resulting patterns are being used for further 
reduction in the next phase. 

Phase-1: Considers restriction of the form 3R.C 
alone, and rule(l) is applied to all the possible pairs in 
the "Rnls. Phase-2: This phase handles restriction of the 
form VR.C, and rules(2-4) are applied in a similar way 
to the resulting set of Phase- 1 . Phase-3: Applies rules(5- 
1 1) to the applicable pairs in the result set of the previ- 
ous phase. Phase-4: Uses rules(12-13). Phase-5: Ap- 
plies rules(14-15) to all possible pairs in the result set of 
Phase-4. If a restriction pair is being reduced (by calling 
Apply-Rule) by a reduction rule r of a particular phase 
p, then those restrictions are removed from the label-set 
while applying the rules in the upcoming phases. 

The reasons for reducing the label-set in phases are: 
(1) the restrictions of the same type can be reduced eas- 
ily, and (2) all the possible reductions can be done by the 
rules given in Table-6. To illustrate our rationale, con- 



an instance with label-set { BhasPet . Creature , 
hasPet . Creature , VhasPet . Owl }. If we are not 
following any particular order for reduction, for n 
number of terms, there are Q ways for choosing 
a pair. In our example, we can take any of the 3 
pairs (BhasPet . Creature , VhasPet . Creature), 
(VhasPet . Creature , VhasPet . Owl) and 
(BhasPet . Creature , VhasPet . Owl) for start- 
ing the reduction. Figure- 1 shows the reduction w.r.t. 
each of the above 3 pairs. In the figure, (1) and (3) 
require additional rules other than the basic reduction 
rules in Table-6, to proceed to a single reduced term, 
and (2) follows our phase based reduction and requires 
no addition rule. 
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(1) 

BhasPet . Creature 

VhasPet . Creature 




(2) 

VhasPet . Creature 

VhasPet . Owl 




SihasPet . Creature 

VhasPet. Owl 




VhasPet . Owl 

BhasPet . Creature 




QhasPet . Owl 



QhasPet . Owl 



(3) 

BhasPet . Creature 

VhasPet . Owl 




ShasPet . Owl 



VhasPet . Creature 



ShasPet . Owl 



Figure 1: Applying reduction rules to pairs of restrictions taken in 
three different orders. (1) and (3) requires additional rule to proceed 
to a single reduced term, and (2) follows our phase based reduction 
order and requires no addition rule. 



Generate-Reduced-Node-Label-Set(L) 

// Input: L = £o(x). 
II Output: R = R(£ Q (x)). 
P = Phases, list of set of pre-defined rules 
R = L\Named-Super-Concepts(L) 
for each phase p e P 
Ri = % 

for each (u,v) e RxR and u + v 

for each rule rep and r applies to (u, v) 
N = Apply-Rule(V, \u, v}) 
R = RU MS)) 
R 2 = R 2 V{u,v] 
R = R\R 2 
return R 



1 

2 
3 
4 
5 
6 
7 
8 
9 
10 
11 



Named-Super-Concepts(L) 

12 S — 0 II Named concepts with named sub concepts 

13 for each named concept a e L 

14 for each named concept b eh where b c a 

15 S = S Ufa) 

16 return S 



A diagrammatic representation of the applica- 
tion of the reduction rules on the restrictions in 
^(XoOiarryPotter)) is shown in Figure-2. Phase- 1 
and Phase-2 rules are not applicable on any of the re- 
strictions in ^(^(harryPotter)). So, the diagram 



shows the reduction phases 3 to 5. In the figure, the pairs 
of restrictions which satisfy the conditions in Table-6 
are reduced based on the corresponding rules. The di- 
rected lines in Phase-3 (in the figure) are paired such 
that each pair represents the applicability of a rule. In 
Phase-4, the restrictions are connected pairwise and the 
arrow head points to the resultant restriction. In Phase- 
5, rules (14-15) are repeatedly applied to find the final 
result. 

3.1.2. Stem-set Generation Heuristics 

Stem-sets are the sets which are used for framing the 
question statements of MCQs. The Rnls of an instance 
is normally used as its stem-set for framing a question. 
Example 1 shows the stem generated from Rnls of 
harryPotter. But there are cases where we cannot 
use a 'Rnls as it is for a stem generation. 

Case 1. A 'Rnls of an instance can contain restrictions 
of the form W?.C (universal quantifier). These univer- 
sal quantifiers are restrictions which may be satisfied 
by the instance under vacuously true condition (oth- 
erwise, they would have been reduced by the reduc- 
tion rules in Table-6). We observed that using such 
restrictions in framing the stem affects the clarity of 
the question. For example, consider the Reduced- 
node-label-set of two instances namely john and bob. 
Let their Reduced-node-label-sets be R(-Co( j ohn)) = { 
Engineer, VhasChild. Doctor } and ??(£o(bob)) = 
{ Engineer, 3hasChild. Doctor } respectively. The 
question framed from the latter Rnls is Choose an En- 
gineer, having only Doctor as child. This stem does not 
mislead the person undertaking the test, since the an- 
swer to the question can be unambiguously chosen as 
bob. This unambiguous selection is because of the fact 
that the linguistic interpretation and the logical state- 
ment (the Rnls) of the stem remains the same. But the 
question generated from the former Rnls misleads the 
test taker, since it asks to choose an engineer whose 
children are all doctors. But, there can be a case that 
j ohn satisfied the restriction VhasChild . Doctor only 
by vacuously true manner. In this case, the linguis- 
tic interpretation of the logical statement under vacu- 
ously true case may confuse the test taker. Therefore, 
we made a design decision to remove such universal re- 
strictions that may be satisfied by vacuously true condi- 
tion from the Rnls to generate the corresponding stem- 
set. So, the stem-set of the instance john becomes 
{ Engineer), and the stem-set of the instance bob re- 
mains the same as that of its Rnls. 



Case 2. The restrictions which contain bottom or 
top concept can be removed from the Rnls while 
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3hasPet.0wl VhasPet. Creature 



3 hasPet.Pet 



< n hasPet. Creature 



Phase 3 




3hasPet.Owl QhasPet. Creature 3hasPet.Pet 3!hasPet. Creature 3!hasPet.Ow 3!hasPet.Pet 




Phase 5 



^^Rule:(14-15) 

3!hasPet.Owl 



Figure 2: Application of reduction rales on the restrictions in *R(X,9(harryPotter)) 
Table 7: Reduced-node-label-sets from Harry-Potter-Book ontology j > 



7?(i^(harryPotter)) 

-R(.£o(ronWeasley)) 

■7?(_£<5(hermioneGranger)) 

7?(-£o(dracoMalfoy)) 

7?(£o(hedwig)) 

<R(£o(errol)) 

■7?(i]c)(scrabbers)) 

*C£o(trevor)) 

■7?(-£<9(crookshanks)) 

■7?(_£<5(nevilleLbottom)) 

^(i^(viktorKrum)) 

^(£o(tomRiddle)) 



= { HogStudent , Wizard, Half Blood, Gryf f indor^^hasPet . Owl } 
= { HogStudent , Weasley, Gryffindor, Pureblood, 3!hasPet . Rat ] 
= { HogStudent , Muggle, Gryffindor, 3!hasPet . Cat } 
= { HogStudent , Slytherin, Pureblood, 3!hasPet.Toad) 
= {0wl, 3isPet0f .HogStudent} 
= {0wl} 

= { Rat , 3isPet0f . HogStudent^ 6 ' 
= {Toad, 3isPet0f .HogStudent } 
={Cat, 3isPet0f .HogStudent } 
—{ HogStudent , Wizard, 3!hasPet .Toad} 
= { Wizard , DrumstrangStud j 
= { Wizard, 



or 



ytherm J 



2, terms of 



generating stem-sets. For example, terms of the form 
3hasChild.T, 3hasPet . ± can be removed from 
the Riils. The terms with bottom or top concept add 
vagueness to the stems of the generated multiple choice 
questions. 

X 

The discussion on how to convert a stem-set to a 
grammatically correct question statement is out of the 
scope of this paper. 

3.1.3. Distractor Generation Procedure 

In the previous section, we have seen that the MCQ 
stems are generated using the stem-set of each of the in- 
stances in an ontology. The instance corresponding to 
the stem-set will become the correct answer. Distrac- 
tors are the set of incorrect options of an MCQ. Given 
an instance x (the correct answer of an MCQ), we use 
its Rnls to identify the set of all instances which can 
be considered as distractors. This set of all possible 
distractors is denoted as D potentia i. We use the below 



condition to determine D potentia i (where d is an instance 
different from x). 



< R{-Co(x)) n Disjoint (Xo(d)) + (f> 



(1) 



To recall from Definition 2, Disjoint set is the set 
which is obtained by taking the union of the set of dis- 
joint concepts of each of the named concepts in the 
node-label-set. Condition- 1 guarantees that there will 
be at least one named concept in the stem forming set 
(stem-set) which is not satisfied by the distractor in- 
stance. This condition also prevents the possibility of 
a distractor becoming a right answer under OWA. We 
call the instance d that satisfies Condition- 1 as a can- 
didate for the distractor of the generated MCQ. The 
procedure Generate-Distractors-Method-1 generates 
possible distractors by giving an instance as input. The 
practicality of generating distractors by making use of 
the disjointness of classes in an ontology is not ques- 
tionable; this is because, Bhmann and Lehmann (2013) 



shows that the axioms which state the disjointness of 
concepts are highly frequent in the ontologies available 
online. 

In Example 1, to find the distractors of the correct 
answer harryPotter, we considered instances like 
hermioneGranger , hedwig etc. which satisfy 
Condition- 1. In the illustration below, we show that 
hermioneGranger and hedwig are not equivalent to 
harryPotter, and they belong to at least one concept 
which is disjoint with a named concept in the stem-set. 

Disjoint (.^(hermioneGranger)) = j Pet , Owl , Rat , 
Toad, Cat, Wizard, Halfblood, Pureblood, 
Slytherin ) 

Disjoint (_£o(hedwig)) = j Rat, Toad, Cat, Muggle, 
Gryffindor, Slytherin, Student, HogStudent , 
Human ) 

7?(-£o(harryPotter)) n Disjoint (.^(hermioneGranger)) 
= ( Halfblood, Wizard ) 

^(^(harryPotter)) n Disjoint (£o(hedwig)) = { 
HogStudent, Gryffindor) 



Generate-Distractors-Method- 1 (x) 
II Input: x, an instance 

II Output: D potentia i, set of possible distractors 

Dpotential — 4> 

for each instance d in O 
if *Uo{x)) n Disjoint {lo{d)) * 4> 

Dpotential = D potential U \d} 

return D potentM 



From the definition of MCQs (Definition 1), let w = 
m - i denotes the count of distractors required for an 
MCQ. We need at least w distractors in the D potentia i set 
of an instance, to consider its finis for MCQ generation. 
Therefore, its only necessary to choose those finis that 
satisfy this condition, for generating stem-sets. In other 
words, for a stem-set, there should be at least w distrac- 
tors in its D p o t e n tiai set - In terms of stem-set, we can 
define node-label-set based MCQs as given below. 

Definition 4. Node-label-set based MCQ. Let Q be 
the multiple choice question with stem-set S, generated 
from fi(£o(x)). The key of Q is defined as K- {x} and 
distractors as D - { y \ (Disjoint (£o(y)) n S ) + (f> 
and y is an instance in O }. 



3.1.4. Difficulty level of MCQs. 

A notion to measure the difficulty level of ontology 
based MCQs was introduced first by Cubric and Tosic 
(2010). Later, Alsubait et al. (2013) extended this idea 
and proposed a similarity-based theory for controlling 
the difficulty of ontology generated MCQs. In (Alsub- 
ait et al. (2012)), the theory has been applied on anal- 
ogy type MCQs. In (Alsubait et al. (2014a)), the au- 
thors have experimentally verified their approach in a 
student-course setup. The practical solution which they 
suggested to find out the difficulty level of an MCQ is 
w.r.t the similarity of the distractors to the key. If the 
distractors are more similar to the key, the students may 
find it very difficult to answer the question, and hence it 
can be concluded that the question is difficult. 

We adopt the above similarity-based theory to cal- 
culate the difficulty level of the generated questions. 
But the conceptual similarity measure (Alsubait et al. 
(2014b)) they have used does not satisfy our require- 
ments. This is mainly because, we are defining MCQs 
and the relationship between instances (key and distrac- 
tors) in terms of finis. In addition, the new notations 
which we introduced (3 and 3!) motivate us to define a 
new similarity measure. The scope of the other similar- 
ity measures in the literature ( Bach and Dieng-Kuntz 
N(2005); Albertoni and Martino (2006); Lehmann and 
Turhan (2012)), is ignored for the same reasons. 

Label-set Similarity Ratio. We introduce a similarity 
measure called Label-set Similarity Ratio (LSR) to find 
out how closely an instance is related to other instances 
w.r.t. their finis. 

Definition 5. Label-set Similarity Ratio (LSR). Given 
two Reduced-node-label-sets U, V; the LSR of U and 
V (denoted by LSR(U,V)j is the ratio of number of ele- 
ments in U which can be associated to some elements in 
V, to the total number of elements in U, i.e., 



LSR(t/, V) 



#{ u | u e U A 3ve V, u ~» v j 
#U 



Association between an element u € U to v € V ( de- 
noted by u v) is based on the conditions mentioned 
below. 

Since we are interested in the Reduced-node-label- 
sets (U and V) of a SHIQ ontology, the elements of 
U (or V) can be a concept name or a term of the form 
3R.C or 3R.C or < nR.C or > nR.C or 31R.C. 

We associate u e U to v e V based on the following 
conditions: 
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1 . A concept u e U is associated to a concept v e V, 
if u E v {concept sub sumption). 

2. If m € i/ is of the form 3R U .C U , we associate it to a 
v ( e V ) of the form 3# V .C V or 3R V .C V or 3!# V .C V 
such that R u E /? v and C„ E C v . 

3. The terms of the form ¥>R U .C U in U are associated 
to the terms of the form 3R V .C V or 3\R V .C V in V 
with R u E and C„ E C v . 

4. If m is of the form < n u R u .C u , we associate it to a v 
of the form < n v R v .C v , such that n v > n u , R u E R v 
and C u E C v . 

5. If u is of the form > n u R u .C u , we associate it to a v 
of the form > n v R v .C v , such that n v < n u , R u E R v 
and C„ E C v . 

6. If m is of the form 3\R U .C U , we associate it to a v of 
the form 3\R Y .C V , such that R u E and C„ E C„. 

For example, consider the 'ftn/s of harryPotter and 
hermioneGranger in Table-7. There are two associa- 
tions from the first label-set to the second. These two as- 
sociations (shown below) are because of the first associ- 
ation condition (concept subsumption). The restriction 
3!hasPet . Owl cannot be associated to 3!hasPet . Cat; 



Definition 6. Given two "Rnls U and V, Closeness is the 
mean ofLSR(U, V) and LSR( V, U). 



Closeness (U,V) 



LSR(t/, V) + LSR(V, U) 



(2) 



The difficulty level of an MCQ (Q) with s as the cor- 
rect answer and {x\ , X2...x w ] (where w + 0) as distractors 
is defined as: 

Difficulty(Q(s, x\, X2...x w )) = 
1 w 

— y cios 



(=i 



iseness 



; ('R(£ 0 (s)),'R(-Co( x d)) (3) 

Considering our previous example, Closeness of the 
Reduced-node-label-sets ^(-^(harryPotter)) and 
^(^(hermioneGranger)) ({/ and V respectively) is 
obtained as followsX. 

LSR(f/, V) = 0.4; LSR(V, U) = 0.5 
Closeness(U, V) = (0.4 + 0.5)/2 = 0.45 

Example 1. Choose a Hogwarts Student, a Wizard, 
ffindor and a Half blood, having exactly 
ne Owl as Pet. 



4 



according to the association condition 6, the former caruAj 
be associated to the latter only if the concept in the lat- 

eGrang 



Options 



Closeness 



ter, Cat, subsumes Owl. 



7?(.C 0 (harryPotter)) 

HogStudent 
Wizard 
Halfblood 
Gryf f indor 
3!hasPet.0wl 



Let U = 7?(i: 0 (harryPotter)) and V = 
7?(.£o(hermioneGranger)); then, #U = 5 and #V 



??(-Co(liermioneGranger)) 

HogStudent 
Muggle 

Gryf f indor 
3!hasPet.Cat 



LSR(£/, V) = 



#{ Gryf f indor, HogStudent } 2 
#U ~ 5 



a. HarryPotter 


1. 


.000 


b. HermioneGranger 


0 


.450 


c. Tom Riddle 


0 


.350 


d. Hedwig 


0. 


.000 


Difficulty 


0. 


.266 




Options 


Closeness 


a. HarryPotter 


1. 


.000 


b. RonWeasley 


0. 


.400 


c. HermioneGranger 


0 


.450 


d. Hedwig 


0. 


.000 


Difficulty 


0. 


,283 



Difficulty level calculation. The difficulty level of a 
generated MCQ can be calculated based on how the 
'Rnls of its distractors are related to the Rnls of the 
correct answer. If the distractors are closely related to 
the correct answer, the difficulty level of the MCQ is (in- 
tuitively) high. The closeness between the two Rnls (U 
and V) is measured in terms of their Label-set Similarity 
Ratio. 



Options 



Closeness 



a. 


Harry Potter 


1. 


.000 


b. 


Draco Malfoy 


0 


.225 


c. 


Ron Weasley 


0 


,400 


d. 


Hermione Granger 


0. 


,450 




Difficulty 


0. 


.358 
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In the above example, we have shown three sets of 
options for the MCQ. The difficulty level of the sets 
are 0.266, 0.283 and 0.358 respectively. Since we as- 
sume that the difficulty level of an MCQ is related to 
the closeness of its options, the third option-set makes 
the MCQ more difficult than the other two sets. Intu- 
itively, the third set of options is indeed the most dif- 
ficult one among all the three sets. This is because, 
the three distractors Draco Malfoy, RonWeasley and 
Hermione Granger are closely related to Harry Potter 
w.r.t. the conditions mentioned in the stem-set. But, 
in the other sets of options an owl named Hedwig is 
appearing as a distracting answer. Hedwig, being an 
odd one among the other options, can be easily ruled 
out as a wrong answer. Clearly, the presence of Hed- 
wig as an option is reducing the difficulty values of the 
MCQs which are generated using the first two sets of 
options. Later in Section 6, we investigate the corre- 
lation between the difficulty levels estimated from our 
Difficulty measure and those determined by domain ex- 
perts for the MCQs generated from a real-world ontol- 
ogy- 

3.2. Edge-label-set Based MCQ Generation 

If an instance x is related to another instance y by a 
role r in O, we can frame a question of (fill in the blanks) < 
type: 

Stem: x r . (Key: y) 

A correct answer (eg: y) can be easily obtained by^en- 
tifying the axioms which can be entailed by the ontol- 
ogy (i.e., O (= r(x,y)). But to select a wrong option (eg: 
z), it does not always mean that O fe£ r(x, z). To guar- 
antee that z is indeed a wrong answer (under OWA), we 
need to entail -ir(x,z). This will ensure the validity of 
the generated question item. By valid question items we 
mean those MCQs whose all distractors are wrong an- 
swers. In what follows, we illustrate a practical solution 
to generate valid question items from edge-label-sets. 

Let us consider the edge-label-set of the pair 
(harryPotter, ronWeasley) from Table-8. 

.^(harryPotter, ronWeasley) = { hasFriend } 

We can generate the stem 

Harry Potter has friend ., 



But the options b, c and d cannot be guar- 
anteed as true distractors, since O |^ 
-ihasFriend (harryPotter , viktorKrum), 
-■hasFriend (harryPotter , tomRiddle) , 
-ihasFriend (harryPotter , hedwig). Therefore, we 
cannot guarantee that this MCQ is valid. A systematic 
method to generate valid MCQs is by considering only 
those roles whose range is limited to some concepts or 
a few instances for stem generation. 

We can identify the roles which are suitable for gener- 
ating the stem of a valid MCQ from a given edge-label- 
sets using the procedure Role-Selection. 

For a given edge-label-set £o(a, b), Role- Selection 
chooses the roles in JLo{a, b) one by one (lines 3-8) and 
checks if they form a part of the restrictions of the form 
3R.C or 3\R.C in the 'Rnls of the instance a. If such a 
restriction exists and if C e -Co(b), then R can be con- 
sidered for framing a stem of the form: a R . 



Role-Selection(£) 



ror 



5 
6 
7 



II Input: E =-Co(a,b), edge-label-set 
// Output: Role sets Pi and P 2 
Pi = P 2 - cf> II Predicate sets 

each role R e E 
for each concept C in O 

if 3R.C e 1l(£o{a)) and C e £o(b) 

Pi = Pi U {R} 
if 3\R.C e ftUo(a)) and C e £o(b) 
P 2 = P 2 v {R} 
return (Pi, P 2 ) 



from the above label-set. Some of 
sible options are a. RonWeasley, 
torKrum, c. TomRiddle, d. Hedwig. 
tion a is a correct answer because of 
tion 



the pos- 
b. Vik- 
The op- 
the asser- 

hasFr iend (harryPotter , ronWeasley) . 



If the role R appears as a part of 3P.C, and C e 
-Co(b), then we store the role R in list Pi (lines 4-5). 
Similarly, the roles which are appearing in the restric- 
tions of the form 3\R.C are stored in P 2 (lines 6-7). We 
handle these two lists separately for MCQ generation. 
The reason for maintaining two lists is explained be- 
low. Table-8 shows list of roles in Pi and P 2 returned by 
Role-Selection, by giving corresponding edge-label-set 
as input. 

3.2.1. MCQ generation from edge-label-set and list P 2 
Consider the edge-label-set .^(harryPotter, 
hedwig) = {hasPet}. From Table-7, we get, 
the restriction 3!hasPet.0wl is an element of 
^(^(harryPotter)). Since the concept Owl in the 
restriction is contained in Xo(hedwig), we can frame 
a stem based on Xo(harryPotter , hedwig) and 
hasPet. Example 2 shows the generated MCQ. 

According to Role- Selection, given 
Xo(harryPotter , hedwig) as input, hasPet 
will get stored in P 2 . Significance of this list is that all 
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Table 8: Some edge-label-sets of Harry-Potter-Book ontology 



Edge-label-set Pi P 2 

_£<9(harryPotter, ronWeasley) = { hasFriend } cf> <p 

.^(harryPotter, hermioneGranger) = { knows , hasFriend, hasHelped) <p <P 

.^(harryPotter , hedwig) = { hasPet } (f> {hasPet; 

£o(trevor, nevilleLbottom) = { isPetOf } {isPetOf} <f> 



the roles of the input edge-label-set which are having 
a strict role restriction are stored in it. For example, 
Harry Potter has exactly one pet which is an owl named 
Hedwig. So intuitively, we only need to choose all other 
instances (including other owls) which are different 
from the instance hedwig as distractors. Therefore, 
we can use the procedure discussed in the previous 
section (Generate-Distractors-Method-1) for finding 
the distractors. 

Given an edge-label-set £o{a,b) with P 2 + <f>, 
we generate distractors by giving b as the input to 
Generate-Distractors-Method-1. The explicit in- 
equalities of individuals in the ontology can be also uti- 
lized in this case as a condition for choosing the dis- 
tractor set. Therefore, Condition (1) which is used in 
Generate-Distractors-Method-1 (line 3) can be modi- 
fied as per the following equation: 



Example 2. Harry Potter has Pet 



Options 



Closeness 



Hedwig 
Crookshanks 
Errol 
Trevor 



Difficulty 



.A 

0.750 
0.500 



0. 500 



0.583 



D 



potential 



= U 



(4) 



(K(XoW)n Disjoint (£o(d)j) 
or xtd 

The relevance of this distractor generation method can 
be explained by the following simple example. In 
the above discussion of Example 2, we have seen that 
hedwig is an (belongs to the concept) Owl; intuitively, 
those Owl instances which are different from hedwig 
form the most apt distractors. Our procedure returns 
such instances also in the possible distractor set, even if 
the inequalities of the instances are not explicitly given 
in the ontology. To elaborate it further, assume that the 
Owl class has two disjoint subclasses: BlackOwl and 
WhiteOwl. hedwig belongs to BlackOwl, and another 
instance browner belongs to WhiteOwl. Also, assume 
that the axiom hedwig # browner, is not present in 
the ontology. Since the restriction 3!hasPet . Owl is on 
Owl, the obvious way to choose the complete distractors 
is by finding all instances which belong to -i0wl. But, 
in Generate-Distractors-Method-1, we consider an in- 
stance as a distractor if r R(£,{x)) n Disjoint (X.(d)) + (p. 
Therefore, the disjointness of classes BlackOwl and 
WhiteOwl is enough to make browner a distractor. 



Options 


Closeness 


a. Hedwig 


1 


000 


b. RonWeasley 


0 


000 


c. Hermione Granger 


0 


000 


d. Scabbers 


0 


500 


Difficulty 


0 


166 



In the above example, we show two sets of options for 
the MCQ. The difficulty levels of the options are 0.583 
and 0.166 respectively. We use the same metrics defined 
in the previous section for calculating Closeness and 
Difficulty. According to those metrics, the first set of 
options makes the MCQ more difficult than the second 
set. Intuitively, since the options in the first set belong 
to some subclass of Creature class, the test takers will 
find it difficult to guess the correct answer (0.25 proba- 
bility for guessing the correct answer). The second set 
contains Ron Weasley and Hermione Granger as two of 
the options. Since those two options can be easily cat- 
egorized as wrong answers, the probability of guessing 
the correct will be increased to 0.5. Hence, the difficulty 
level of the MCQ which uses the second set of options 
should be less when compared to the MCQ generated 
with the first set. 

3.2.2. MCQ generation from edge-label-set and list P\ 
We consider the edge-label-set 

Xo(trevor, nevilleLbottom) in Table-8 for 
our illustration. The list Pi of this label-set contains 
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the role isPetOf . This means that the ^(^(trevor)) 
contains a restriction of the form 3isPet0f.C 
(where C is some concept). From Table-7, we get, 
SisPetOf .HogStudent e ^(i^(trevor)). This 
restriction on isPetOf is not that strict when compared 
to the restriction in the previous case (3!hasPet . Owl). 
3isPet0f .HogStudent states that there should be at 
least one isPetOf relation from the instance trevor, 
and the range of the relation should be HogStudent. 
Therefore, when we frame an MCQ using the edge- 
label-set £o(trevor ,nevilleLbottom), the only 
possible way to choose a distractor is by finding the 
instances in the complement of the class HogStudent. 
Generate-Distractors-Method-2 is designed to satisfy 
this requirement. Example 3 is generated using our 
example edge-label-set and the role isPetOf . 

Generate-Distractors-Method-2(C) 

II Input: C, the restriction class of the role 
II Output: D potential, list of possible distractors 

1 ^potential = <P 

2 for each instance d in O 

3 if C e Disjoint (£o(d)) 

4 Add d to D potent i a i 

5 return D 'potential 



Example 3. Trevor is the pet of 



Options 



Closeness 



a. 


Neville Lbottom 


1. 


.000 


b. 


Viktor Krum 


0 


,415 


c. 


Hedwig 


0. 


.000 


d. 


Errol 


0. 


.000 



Difficulty 



— 



.38 



Options 



Neville Lbo ttom 
Hedwig 



a. 
b. 

c. Crooksha: 

d. Scabbers 



nks 



1.000 
0.000 
0.000 
0.000 



Difficulty 0 . 000 



In the above example, the first set of options makes 
the MCQ more difficult than the second set of options. 
The Closeness values of some options are zero because 
of the fact that the Reduced-node-label-sets of such op- 
tion instances cannot be related to the Unls of the correct 
answer instance (we consider those instances as unre- 
lated ones). 



4. Related Work 



Papasalouros et al. (2008) suggested 11 strategies 
based on classes, properties and terminologies of on- 
tologies for framing MCQs and the corresponding dis- 
tracting answers. Their MCQ generation methods based 
on these strategies lack proper theoretical backing to 
support when to use which strategy, and the stem of all 
the generated questions remains the same ( "Choose the 
correct sentence"). 

Cubric and Tosic (2010); M.Tosic and M.Cubric 
(2009) generated MCQs of knowledge level {"Which 
of the following definition describes the concept C?"), 
comprehension level {'Which one of the following re- 
sponse pairs relates in the same way as a and b in the 
relation /??'), application level {"Which one of the fol- 
lowing examples demonstrates the concept C?") and 
analysis level ( "Analyze the text x and decide which one 
of the following words is a correct replacement for the 
blank space in x."). Their work is an extension of the ap- 
proach by Holohan et al. (2005), by introducing stems 
that use annotation information in the ontology. Strate- 
gies similar to Papasalouros's strategies are adopted to 
find the distracting answers (distractors) for the gener- 
ted MCQs. 

Another MCQ generation method is by Alsubait et al. 
(2012). They present an approach called similarity- 
based approach for generating analogy type questions. 
In their question generation algorithm, a set of parame- 
ters are introduced to control the difficulty level of the 
generated questions. They argue that the difficulty level 
of a question item can be increased by finding the dis- 
tractors which are similar to the correct answer(s). The 
approach which the authors illustrate is limited to anal- 
ogy type questions. 

Other than the above MCQ generation approaches, 
there are works like Abacha et al. (2013); Ben Abacha 
and Zweigenbaum (2012); Aitko et al. (2009), which 
make use of simple ontology statements: concept inclu- 
sions, role hierarchy and (concept and role) assertions, 
to generate questions of basic types. 

In addition to the above MCQ generation approaches, 
a few researchers worked on rule-based methods for 
question answer generation. The work by Zoumpa- 
tianos et al. (2011) uses Semantic Web Rule Language 
(SWRL), a combination of OWL with RuleML 4 , to gen- 
erate MCQs. 



4 http://wiki.ruleml.org/index.php/RuleML_Home (last accessed 
11th Dec 2014) 
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Table 9: MCQ examples generated from People & Pet ontology using existing approaches 



1 . Which one of the following response pairs is related in the 
same way as Walt and Huey in the relation hasPetl 

a. Joe hasPet Fido (T) 

b. Fred hasPet Huey (F) 

c. Walt hasPet Louie (F) 

d. Minne hasPet Tibbs (F) 



3. Choose the correct answer. 

a. Fido is a dog (T) 

b. Tom is a dog (F) 

c. Fluggy is a dog (F) 

d. Louie is a dog (F) 



4. Choose the correct answer. 

a. Man is a Person (T) 

b. Cat is a Person (F) 

c. Duck is a Person (F) 

d. Giraffe is a Person (F) 



2. Choose the correct answer. 

a. Walt hasPet Fido (T) 

b. Minne hasPet Fido (F) 

c. Minne hasPet Dewey (F) 

d. Kevin hasPet Louie (F) 

5. Haulage Truck Driver : Driver 

a. Bus : Vehicle (T) 

b. Quality Broadsheet : Newspaper (F) 

c. Giraffe : Sheep (F) 

d. Giraffe : Cat Liker (F) 



5. Experiments and Comparative Study 

In order to support the approaches presented in this 
paper, we implemented our techniques in Java 1.7 using 
the Jena framework (2.11.0) as a portable library. We 
use the Jena Ontology API 5 to explore concepts and to 
look up restrictions in the ontologies. 

We consider four ontologies for our experimentation. 

1 . Harry-Potter-Book ontology 6 is developed by our re- 
search group at IIT Madras, India. 



Table 10: Specifications of ontologies which are used for experiments 

Classes Obj -Props Indvs 



Ontology 



— 



Harry-Potter-Book 
People & Pet 
Plant-Protection 

hical Entity 




17 


5 


12 


60 


15 


21 


105 


15 


546 


34 


4 


306 



^ to make them suitable to generate expected results. 
2. People & Pet ontology 7 is developed by Sean BecfcJO In GEO ontology, we have added axioms to state that 
hofer et al. for ISWC 2003 tutorial on OWL. the concepts: 'group of dependencies', 'group 



3. Plant-Protection ontology* models knowledge about 
cereal-plant protection; it was developed by Interna- 
tional Center of Agricultural Research in the Dry Ar- 
eas, University of Aleppo, Syria^ 

4. Geographical Entity ontology 9 (GEO ontology) is 
based on geopolitical organizations and their subdivi- 
sions. This ontology is developed by William Hogan, 
University of Florida, US. 

The specifications of the four ontologies are pre- 
sented in Table- 10. Some necessary disjointness 
between concepts were not there in the ontologies (2) 
and (4). So, we added a few axioms to those ontologies 



5 http://jena.apache.org (last accessed 11th Dec 2014) 
6 https://sites.google.com/site/ontoworks/ontologies (last accessed 
11th Dec 2014) 

7 http://www.cs.man.ac.uk/~horrocks/ISWC2003/Tutorial/people 
+pets.owl.rdf (last accessed 1 1th Dec 2014) 

8 https://sites. google.com/site/ppontology/ (last accessed 1 1th Dec 
2014) 

9 http://repos.frontology.org/geographical-entity-ontology/src (last 
accessed 11th Dec 2014) 



the concepts: 

of major administrative subdivision', 
nation, 'geopolitical dependency' and 
'major administrative subdivision' are 
disjoint. We added axioms to People & Pet ontology 
to state that cat , duck, giraffe, person, sheep 
and tiger are disjoint concepts. We also included the 
axiom (male n female E _L) and the axioms to state 
that publication, vehicle and animal are disjoint 
concepts. Three assertions: Male(joe), Male(walt) 
and Male (f red), are added to the ontology, to gener- 
ate distractors which are closely related to the correct 
answer. 

For a comparative study, in Table-9, we present sam- 
ple MCQs which are generated from the People & Pet 
ontology by using the existing approaches. Each op- 
tion is marked with (T) or (F) to represent the key and 
the distractor respectively. Question 1 in the table is 
generated using the approach proposed in Cubric and 
Tosic (2010); M.Tosic and M.Cubric (2009). Questions 
2 to 4 are generated using class, property and terminol- 
ogy based strategies (respectively) described in Papasa- 
louros et al. (2008). Question 5 is an analogy type ques- 
tion from the approach in Alsubait et al. (2012). Exam- 
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Example 4. Choose a person and Example 5. Minne has pet 
an oldlady, having only cat 
as pet. 



Example 6. Joe has pet 



Options 


Closeness 


Options 


Closeness 


Options 


Closeness 


a. Minne (T) 


1.000 


a. Tom (T) 


1.000 


a. Fido (T) 


1.000 


b. Joe (F) 


0.375 


b. Louie (F) 


0.750 


b. Louie (F) 


0.750 


c. Walt (F) 


0.458 


c. Fido (F) 


0.750 


c. Tom (F) 


0.750 


d. Fred (F) 


0.550 


d. Joe (F) 


0.000 


d. Huey (F) 


0.750 


Difficulty 


0.461 


Difficulty 


0.500 


Difficulty 


0.750 



Example 7. Choose a White 
a van and a white thing, 
some cat as pet. 



van man and a dog owner, drives 
reads only tabloid, having 



Options 



Closeness 



a. 


Mick 


(T) 


1. 


.000 


b. 


Fred 


(F) 


0, 


.392 


c. 


Fido 


(F) 


0. 


,000 


d. 


Joe 


(F) 


0, 


,547 



Difficulty 0.313 




Example 8. Choose a dog owner, 
having exactly one dog as pet . 



Options 



(T) 
(F) 
(F) 
(F) 



Closeness 

1.000 
0.000 
0.375 
0.000 



Difficulty 



0.125 



pies 4-8 show sample MCQs which are generated using 
our approaches from the same ontology. 

Table- 1 1 shows the count of the MCQs generated per 
ontology using our two approaches. Sample questions 
generated from GEO ontology are given in Appendix A. 
A detailed list of MCQs generated by our approaches 
is available at an external source 10 . It should be noted 
that, even though we generate large number of MCQs, 
the stems of many of these MCQs are the same. This is 
because, an ontology can have many instances with the 
same ftnls. 

G° 

6. Empirical Evaluation 



In this section, we describe the empirical evaluation 
done to (1) check the usefulness of the question items 
which are generated from our two MCQ generation ap- 
proaches and (2) verify the effectiveness of the proposed 
measure for estimating question difficulty. 



10 https://docs. google. com/document/d/10tkswD7- 
Z7hcBsUroxWhzuY0zse5SUSYt"KSbVI4T2Lk/edit?usp=sharing 
(last accessed 21th Dec 2014) 



6.1. Ontology and question-set 

We consider MCQs generated from Plant-Protection 
ontology (PP ontology) for our empirical evaluation. A 
total of 656 questions have been generated from the on- 
tology. Manual evaluation of all the generated questions 
is difficult. We grouped the questions which are based 
on the same label-set and, randomly selected one ques- 
tion from each group to form a question-set. A question- 
set of 31 questions is generated from the ontology for 
evaluation. 

Participants of the evaluation are seven experts of 
plant disease or agriculture related domains. Four of the 
participants have PhD and three have masters degree in 
the plant protection related areas. Further details of the 
participants are given in the acknowledgement section. 

6.2. Evaluation method 

We present the participants with the prepared 
question-set. Each question in the question-set is pro- 
vided with three check boxes along with the following 
guidelines: 

□ Useful Select this option if the stem (question 
statement) of the MCQ is useful in conducting a domain 
related assessment test. 
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Choose a plant fungal disease, having only some microorganism as factor. 

□ Useful □ Not useful, but domain related □ Not useful and not domain related 



a. Scald (T) 

b. Black Chaff (F) 

c. Bacterial Leaf Blight (F) 

d. Barley Yellow Dwarf Viroses (F) 

Hardness: □ High 

□ Medium 

□ Low 



a. Scald (T) 

b. Black Chaff (F) 

c. Bacterial Leaf Blight (F) 

d. Nitrogen Excessiveness (F) 

Hardness: □ High 

□ Medium 

□ Low 



a. Scald (T) 

b. Black Chaff (F) 

c. Nitrogen Deficiency (F) 

d. Nitrogen Excessiveness(F) 

Hardness: □ High 

□ Medium 

□ Low 



Figure 3: A sample question from PP ontology which is used for empirical study 



□ Not useful, but domain related Select this option 
if the stem is not useful in conducting a domain related 
assessment test, but it is related to the domain. 

□ Not useful and not domain related Select this 
option if the stem is neither useful nor related to the 
domain. 

They need to select one of the above check boxes. 
This helps in identifying the usefulness of the generated 
MCQs. 

Each question has a set of answer options. To check 
the quality and hardness of the option-set we provide 
three check boxes: □ High □ Medium and □ Low. 
The participants need to tick one among the three check^J 
boxes. Some of the question items may contain more 
than one set of option-sets. In that case, they need to 
relatively grade the option-sets. A sample question from 
the question-set is given in Figure-3. 

A question which has been rated as Useful by at least 
four out of the seven participants are considered to be 
a real-useful question for conducting assessment tests. 
The hardness score of the option-sets are determined 
from the participant's rating in the same way, by consid- 
ering the majority. We compare the obtained hardness 
rating with the pre-calculated hardness which is based 
on our Difficulty measure, to judge the effectiveness of 
our measure. £^ 



is, usi 



While calculating the hardness, using Difficulty mea- 
sure, a question item is given a low score if the calcu- 
lated difficulty is in the range 0 to 0.33. Similarly, a 



score of medium and 
in the range 0.33<^C 

6.3. Results^?* 




s given, if the difficulties are 
6 and 0.66< to 1.0 respectively. 



Table 11: No. of questions generated by proposed approaches 

Approach based on 



Ontology 



node-label-set edge-label-set 



Harry-Potter-Book 
People & Pet 
Plant-Protection 
Geographical Entity 



12 
21 
564 
305 



8 
2 
92 
0 



From Table- 12, it is evident that a large number of 
questions in the question-set are useful in conducting 
a domain related assessment test. Out of the 23 ques- 
tions which are generated using node-label-set based 
approach, the stems of 8 questions contain role restric- 
tion constraints (like: "having only some microorganism 
as factor"). Majority of the participants have rated all 
these 8 questions as useful. The question item with stem 
"Choose an air pollutant." is the only question which is 
marked as Not useful, but domain related. Some ques- 
tions like "Choose a drainage." are categorized as Not 
useful and not domain related ones. Therefore, there is 
a need to screen the generated MCQs prior to adminis- 
trating them on a real-world assessment test. Our analy- 
sis of the results and oral feedback from the participants 
show that the MCQs with role restrictions are very use- 
ful and realistic in conducting tests. 

The effectiveness of the proposed Difficulty measure 
is studied by finding out the number of option-sets 
whose hardness scores are same when (1) calculated us- 
ing Difficulty measure, and (2) determined by domain 
experts. In total, we used 75 option-sets (since most 
of the questions in the question-set have more than one 
option-sets) for our evaluation. 

In Table- 13, the first row shows the number of option- 
sets that are rated high, medium and low using the pro- 
posed difficulty measure. The second row shows what 
the domain experts determined. The last row shows, for 
each category, the number of option sets that were cor- 
rectly identified as belonging to that category by both 
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Table 12: Statistics to find the usefulness of questions generated from PP ontology 



Approach 






Count of questions 


Total 


Useful 


Not useful, but domain related Not useful & not domain related 


Node-label-set based 
Edge-label-set based 


23 
8 


16 
6 


1 6 
0 2 



the proposed measure and domain experts. We ob- 
serve that the hardness scores of 49 out of 75 option- 
sets (nearly 65%) are correctly calculated by our Dif- 
ficulty measure. This highlights the employability of 
the proposed measure in determining the hardness of 
the generated MCQs. In the cases where the calculated 
score deviated from the predicted score, we observed 
that, irrespective of the similarity of the distractors with 
the key, there are clues in the correct answer instances 
which help in identifying the key easily, making the 
hardness of the MCQ as low. One such example MCQ 
is given below, where the hardness score is calculated as 
Medium, but the domain experts have given a Low score 
because of the word "Virus" in the key. 

Example 9. Choose a virus. 



Options 



Closeness 



a. 


Barley Stripe Mosaic Virus 


(T) 


1. 


,000 


b. 


Pyrenophora Graminea 


(F) 


0. 


,500 


c. 


Xanthomonas sp. 


(F) 


0. 


,500 


d. 


Sclerophthora Rayssiae 


(F) 


^1 


,500 



Difficulty *SS0. 500 



Table 13: Statistics to find the effectiveness of Difficulty measure 
No. of option-sets with hardness 
High Medium Low Total 



Proposed ^ 
Difficulty Meas-- 



Domain Expe: 
Opinion 



asure 
Ss— 



23 



29 75 



12 



17 



46 75 



No of option sets 
correctly classified 
by the proposed 
measure. 



10 



13 



26 49 



The plant-protection question-set and the response- 
sheets of the participants are available at our project 
web-page 11 . 



11 https://sites.google.com/site/ontoworks/projects 



7. Conclusion and Future Work 

We propose the generation techniques of two new cat- 
egories of multiple choice question types based on the 
node-label-sets and edge-label-sets of the instances in 
an ontology. We introduced a technique called Label- 
set-Reduction to make the label-sets suitable for gener- 
ating MCQs by converting them to a more reduced form 
(called Reduced-node-label-sets). We suggested a sys- 
tematic method to find the distractors for the generated 
MCQs. The similarity of these distractors with the cor- 
rect answer is determined using a closeness measure, 
which is in turn used to estimate the hardness level of 
the MCQs. We considered the open-world assumption 
and DL semantics in our approaches to make the MCQs 
valid and realistic. The feasibility of the approaches 
are studied by experimenting them on a synthetic ontol- 
ogy and on three ontologies which are available online. 
Our experiments prove that the proposed approaches, 
when used with semantically-rich ontologies, can pro- 
vide successful cases. We also did an empirical study 
by generating question items from a real-world ontol- 
ogy and the domain experts' evaluation of the generated 
MCQs is encouraging. 

While the proposed approaches work well in defin- 
ing the semantics of questions, the problem of gener- 
ating syntactically correct question items is only par- 
tially tackled. In order to overcome syntactic problems, 
more sophisticated natural language generation tech- 
niques should be utilized in future implementations of 
the presented approaches. 

In this paper, we assumed that the difficulty level of 
an MCQ is purely dependent on how closely its cor- 
rect answer is related to its distractors. We considered 
all stems with same level of importance and assumed 
that the option-set makes one question more difficult 
(or easier) than the other. Nevertheless, the question 
statement itself can make an MCQ difficult (or eas- 
ier). For example, consider the stem of Example 2 
with stem "Harry Potter has pet .", we can de- 
crease the difficulty level of this question by adding the 
class information of the pet of Harry Potter in the stem 
( "Harry Potter has pet , an Owl"). 
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Another observation we made during our work is 
about the relevance of the generated MCQ w.r.t. the do- 
main of the ontology. For example, consider the ques- 
tion "Choose a drainage." which is generated from 
Plant-Protection ontology. This question is categorized 
as not useful in conducting assessment tests and not re- 
lated to the domain by domain experts. Similarly, when 
considering the questions (given in Appendix A) gen- 
erated from GEO ontology, Example 5 appears to be a 
less relevant MCQ (when compared with Examples 1 
to 4 in Appendix A), since its stem "Choose a nation." 
is less related to the domain: geopolitical organization, 
but more related to a generic ontology domain which 
talks about countries, continents and nations. An au- 
tomated method for finding the relevant concepts and 
roles in an ontology is required to enhance the applica- 
bility of the proposed approaches. 
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Appendix A. MCQ Examples from GEO ontology 



Example 5. Choose a nation. 



Sample MCQs generated from Geographical Entity 
Ontology using node-label-set and edge-label-set based 
approaches are given below: 

Example 1. Choose a geopolitical dependency, 
a member of exactly one sovereign state. 



Options 



Closeness 



a. 


India 


(T) 


1. 


.000 


b. 


Bermuda 


(F) 


0. 


.000 


c. 


Aruba 


(F) 


0. 


.000 


d. 


Jersey 


(F) 


0. 


.000 



Difficulty 0 . 000 



Options 



Closeness 



a. 


Jersey (T) 


1. 


,000 


b. 


Illinois (F) 


0. 


.500 


c. 


Maryland (F) 


0. 


.500 


d. 


Hawaii (F) 


0. 


.500 




Difficulty 


0. 


.500 



Observe that, difficulty level for examples 4 and 5 is 
zero, this is because GEO ontology is not semantically 
rich enough to relate the correct instance and the dis- 



tractor instances. 



Example 2. Choose a geopolitical dependency, 
a member of exactly one sovereign state. 



Appendix B. Notations 

label-set 



Options 




Closeness 


a. Anguilla 


(T) 


1.000 


b. Massachusetts 


(F) 


0.500 


c. Lowa 


(F) 


0.500 


d. Uruguay 


(F) 


0.000 




Difficulty 


0.333 



£o(x) 
-Co(x,y) 



Example 3. Choose a major administrative 
subdivision, a part of some nation and 
a member of exactly one sovereign state. 



Node-label-set of instance x in ontology O 
Edge-label-set of instances x,y in ontology 

'Ri-Co (x)) Reduced-node-label-set of x in ontology O 
3\R.C 3R.C n <1R.C 
3R.C nVfi.C 
Reduced-node-label-set 
Label-set Similarity Ratio 
D 'potential Set of possible distractors 



3R.C 
fa ftnls 
LSR 



Options 


Closeness 


a. Florida (T) 


1.000 


b. Aruba (T) 


0.500 


c. Bermuda (F) 


^500 


d. Guam (F) 


<Qo* 500 


Difficulty 


0.500 



Example 4. Choose a group of major admini- 
strative subdivision. 



Options Closeness 



a. 


US State 


(T) 


1. 


.000 


b. 


US Territories 


(F) 


0. 


.000 


c. 


US Commonwealths 


(F) 


0, 


.000 


d. 


Israeli-occupied Territories 


(F) 


0. 


,000 






Difficulty 


0. 


.000 
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